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ABSTRACT 

Uncertainty  can  significantly  affect  a  system’s  cost.  Any  estimate/prediction  of  a  system’s 
cost  is  accompanied  by  a  range  of  other  potential  costs.  It  is  not  overly  pessimistic  to  assign  at 
most  a  30  percent  confidence  level  to  a  complex  system’s  cost  estimate  to  account  for  significant 
technical  and  program  uncertainties,  and  the  estimation  uncertainty  itself.  A  decision-maker 
must  understand  the  uncertainties  and  their  potential  impact  while  weighing  these  uncertainties 
against  their  cost  and  his  resource  constraints. 

Decision  analysis  provides  a  structured  approach  for  analyzing  and  selecting  a  course  of 
action  to  achieve  an  objective  where  the  choices  and  outcomes  are  characterized  by  uncertainty 
as  well  as  the  value  tradeoffs.  For  difficult  decisions  with  careful  analysis  and  hard  choices,  it  is 
human  nature  to  still  wonder  if  you  made  the  right  decision.  Regret  analysis,  a  subset  of  decision 
analysis,  is  a  basic  operations  research  technique  used  to  determine  how  much  may  be  lost  if  a 
different  decision  were  chosen.  Regret  occurs  when  the  selected  alternative  is  not  the  most 
preferred  outcome  (e.g.  the  least  expensive).  This  paper  illustrates  an  application  of  Regret 
Analysis  via  a  case  study  using  Head  Quarters  Air  Force  Space  Command’s  recent  Operationally 
Responsive  Spacelift  Analysis  of  Alternatives. 

Introduction 

This  paper  defines  the  difference  between  “risk”  and  “regret”.  It  describes  a  variation  of 
regret  analysis  for  the  purposes  of  a  case  study,  and  highlights  the  different  analysis  outcomes 
between  the  two  types  of  regret  analysis  accomplished  in  the  case  study. 

To  illustrate  this  difference,  this  paper  will  explore  the  Head  Quarters  Air  Force  Space 
Command  (HQ  AFSPC)  Operationally  Responsive  Spacelift  (ORS)  Analysis  of  Alternatives 
(AoA)  as  a  case  study.  The  purpose  of  the  ORS  AoA  was  to  determine  the  most  responsive 
means  to  launch,  maneuver,  service,  and  retrieve  space  payloads  to  enhance  military 
effectiveness.  The  ORS  AoA  was  designed  to  provide  United  States  Air  Force  and  Department 
of  Defense  with  critical  information  about  the  military  effectiveness  and  cost  for  making 
acquisition  decisions  concerning  ORS.  The  main  questions  for  the  ORS  AoA  to  answer  are: 

•  What  are  the  benefits  of  acquiring  ORS? 

•  Is  ORS  the  best  way  to  achieve  these  benefits? 
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•  Is  ORS  affordable? 

•  If  so,  what  is  the  best  approach  to  develop  ORS? 

Decision-makers  involved  with  ORS  range  from  the  HQ  AFSPC  Commander  -  responsible 
for  the  development,  acquisition  and  operation  of  the  Air  Force's  space  and  missile  systems,  to 
the  acting  Secretary  and  Under  Secretary  of  the  Air  Force,  Washington  D.C.  and  Department  of 
Defense  Executive  Agent  for  Space.  These  decisions-makers  are  faced  with  a  choice  among  a 
wide  variety  of  launch  vehicles  (LVs)  offering  a  range  of  performance  and  cost.  The  ORS  AoA 
identified  a  launch  vehicle  architecture  that  on  the  basis  of  cost  and  risk  appears  most  worthy  to 
pursue. 


Risk  vs  Regret  Defined 

Formally  speaking,  risk  is  the  probability  an  unfavorable  event  occurs.  Risk  can  be  defined 
as  the  chance  of  potential  harm  that  may  arise  from  some  present  or  future  action.  It  is  often 
defined  as  the  probability  of  an  undesirable  event.  Probabilities  and  value  assessments  of  the 
consequences/impacts  of  events  are  combined  into  an  expected  outcome. 

Risk  is  also  defined  as  an  adverse  event  or  outcome,  foreseen  or  not,  that  impacts  an 
organization,  operation,  process,  or  project  in  a  negative  way.  Although  risk  has  no  one 
definition,  some  theorists,  notably  Ron  Dembo,  have  defined  general  methods  to  assess  risk  as  an 
expected  after-the-fact  level  of  regret. 

Webster’s  Revised  Unabridged  Dictionary  (1913)  defines  regret  as  “To  experience  regret  on 
account  of;  to  lose  or  miss  with  a  sense  of  regret;  to  feel  sorrow  or  dissatisfaction  on  account  of 
(the  happening  or  the  loss  of  something);  as,  to  regret  an  error;  to  regret  lost  opportunities  or 
friends.”  In  game  theory,  the  key  idea  is  to  minimize  regret  -  the  feeling  one  gets  after  making  a 
bad  decision  and  wishing  the  decision  could  be  changed  after  the  game  is  finished.  The  end 
product  of  this  line  of  reasoning  is  that  one  would  make  decisions  in  such  a  way  as  to  minimize 
the  maximum  expected  loss  to  himself  or  herself. 

In  addition,  regret  associated  with  an  outcome  is  classically  defined  as  the  utility  of  that 
outcome  subtracted  from  the  maximum  utility  of  that  state  (condition)  -  in  other  words,  the 
amount  of  additional  satisfaction  which  one  could  have  if  they  made  a  better  choice.  When  the 
utility  of  an  outcome  is  the  maximum  for  that  state  (condition),  the  regret  associated  with  that 
outcome  equals  zero.  Regret  measures  the  extent  to  which  one  would  be  unhappy  to  miss  out  on 
a  positive  outcome  (or  opportunity).  It  also  expresses  how  much  worse  things  are  than  they 
would  have  been  had  one  chosen  the  other  option  within  a  certain  state  (condition). 

A  variation  of  regret  analysis  was  applied  in  the  case  study  described  in  this  paper  where 
alternatives  offering  the  same  utility  at  a  defined  level  are  compared  and  “regret”  of  an 
alternative  is  measured  in  terms  of  the  difference  in  life  cycle  cost  (LCC) — the  lost  opportunity 
for  not  selecting  the  least  expensive  alternative.  The  same  alternatives  are  compared  at  three 
different  defined  utility  levels  with  their  relative  cost  and  risk  of  achieving  those  levels  not 
consistent  across  the  levels. 

A  Real  Example  -  ORS  AoA  Case  Study 

Introduction:  The  regret  analysis  case  study  is  based  on  the  ORS  AoA.  The  Under 
Secretary  of  Defense  for  the  Air  Force  (Space)  commissioned  HQ  AFSPC  Directorate  of 
Requirements  (AFSPC/DR)  to  deliver  an  AoA  to  determine  the  best  means  to  responsively 
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launch,  maneuver,  service  and  retrieve  space  payloads  to  enhance  military  effectiveness.  The 
team  took  a  capabilities-based  approach  rather  than  a  traditional  AoA  approach  of  identifying  the 
alternatives  and  then  determining  military  utility.  Typically  an  AoA  defines  and  assesses  a  set  of 
alternatives  for  cost  and  effectiveness,  using  defined  measures  (Measures  of  Effectiveness 
(MOEs))  for  meeting  mission  needs.  Effectiveness  is  determined  through  a  utility  analysis  that 
relates  measures  of  system  performance  (MOPs)  to  MOEs.  By  contrast,  the  ORS  AoA  began  by 
establishing  several  levels  (low,  medium,  and  high)  of  mission  utility  and  corresponding 
performance  levels  to  which  alternative  designs  were  defined  and  subsequently  costed.  The 
advantage  of  this  approach  is  that  performance  levels  are  fixed  enabling  the  alternatives  to  be 
evaluated  and  compared  strictly  on  the  basis  of  cost  and  risk.  The  variation  on  regret  analysis 
described  in  this  paper  was  applied  to  enable  this  comparison. 


Figure  1  represents  the  capabilities-based  approach  of  the  ORS  AoA. 
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Figure  1.  The  ORS  AoA  Process. 


The  ORS  AoA  entailed  a  complex  senes  of  analyses  performed  by  several  working 
groups.  Detailed  descriptions  of  each  analysis  are  beyond  the  scope  of  this  paper.  The  military 
utility  analysis  conducted  by  the  ORS  AoA  Effectiveness  Analysis  Working  Group  encompassed 
a  series  of  parametric  experiments  using  the  Lightning  campaign  model  to  establish  the  three 
levels  of  performance  (low,  medium,  and  high),  and  their  associated  utility. 

Payload  Analyses  and  Mission  Task  Analyses  (Maintaining  On-orbit  capability  in  the 
presence  of  Red  Offensive  Counterspace  (OCS),  Blue  Offensive  Counterspace,  Force 
Application)  were  used  to  define  the  low,  medium,  and  high  performance  parameters.  Table  1 
provides  a  summary  of  the  ORS  AoA  Effectiveness  Analysis  Working  Group’s  definition  of  low, 
medium,  and  high  performance  used  for  defining  and  evaluating  ORS  alternatives. 
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Parameters/U  tility 
Levels 

Low 

Medium 

High 

Red  OCS 

50% 

(reduction  in  Blue 
capability) 

85% 

(reduction  in  Blue 
capability) 

100% 

(reduction  in  Blue 
capability) 

Blue  Replenishment 

50% 

(end  result  of  75%  of 
original  capability) 

70% 

(end  result  of  75%  of 
original  capability) 

85%  of  original  capability 

Blue  OCS 

None 

Attacks  Red  NAV  at  50% 
on  day  2 

Attacks  Red  ISR  at  100% 
on  day  0  and  NAV  75%  or 
day  0 

#  of  Common  Aero 
Vehicles  (CAVs) 

50  in  20  days 

500  in  10  days 

2000  in  20  days 

Table  1:  Overall  Utility  Settings  for  Low,  Medium,  and  High 


(Note:  The  start  of  a  Major  Theater  War  is  Day  0.  D-0  represents  Day  0,  D-2  represents  Day  0  minus  2  days) 


To  develop  alternatives  for  the  ORS  AoA,  the  Integrated  Concepts  Working  Group  (ICWG) 
first  defined  specific  payload  (e.g.,  satellite/constellation,  weapon  system)  parameters  and 
required  deployment  timeframes  to  meet  mission  needs.  The  ICWG  established  launch 
schedules  using  that  data  and  then  defined  “launch  vehicle  architectures”  (LVAs)  to  meet  the 
demands  of  the  launch  schedules.  LVs  were  grouped  into  LVAs  so  that  each  LVA  had  sufficient 
capacity  to  launch  all  space  assets  being  defined.  An  LVA  can  have  one,  two,  or  three  launch 
vehicles  in  its  architecture.  The  ICWG  then  costed  each  of  the  LVAs  for  each  performance 
level,  reflecting  the  different  mix  and  quantity  of  launch  vehicles  needed  within  the  architecture 
to  achieve  the  specific  performance  level.  With  each  LVA  having  the  same  performance 
capability  at  a  given  performance  level,  only  cost  and  risk  differentiated  them. 

The  results  of  the  Military  Utility  Analysis  and  Alternative  Development  Phases  of  the  ORS 
AoA  resulted  in  a  large  number  of  alternatives  to  meet  three  different  performance  levels, 
yielding  over  85  LVAs.  For  detailed  analysis  the  ICWG  reduced  the  set  to  15,  spanning  the  four 
types  of  vehicles:  3  Expendables,  8  Reusables,  3  Hybrids,  and  1  Hypersonic.  Table  2  shows  the 
set  of  15  architectures  which  meet  the  required  performance  levels.  (Note:  This  paper  will  not 
discuss  the  details  of  the  LVs  in  Table  2  but  rather  show  the  reader  the  four  LVA  groups  and  the 
LVs  which  make  up  the  specific  LVAs.) 
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LVA# 

Configuration 

Vehicle  A 

Vehicle  B 

Vehicle  C 

Exp  LVAs 

4 

Sol  25  k 

Exp-Generic  EELV  45  k 

It 

RPRP  25  k 

Exp-Generic  EELV  45  k 

12 

RPRP  45  k 

Reuseable  LVAs 

30 

RPOpt  15k 

RPOpt  Growth  to  Trimese 

36 

RPLHOpt  5  k 

RPLHOpt  Growth  to  45  k 

39 

RPLHOpt  25  k 

Exp-Generic  EELV  45  k 

41 

RPLHOpt  45  k 

46 

LH  Bimese  15  k 

LHBi  Growth  to  45  k 

47 

LH  Bimese  1 5  k 

LHBi  Growth  to  Trimese 

48 

LH  Bimese  25  k 

Exp-Generic  EELV  45  k 

51 

LH  Bimese  45  k 

Hypersonic  LVAs 

64 

RBCC  25  k 

Exp-Generic  EELV  45  k 

Hybrid  LVAs 

67 

RLV/ELV-Solid  RD180 

Exp-Generic  EELV  25  k 

Exp-Generic  EELV  45  k 

70 

RLV/ELV  12.7k 

Hybrid  2  RLV/ELV  45k 

71 

RLV/ELV  19.5k 

Hybrid  Growth  to  Trimese 

Table  2:  Final  ORS  AoA  Launch  Vehicle  Architectures 


In  addition,  seven  different  Space  Architectures  (SAs)  were  developed  within  which  the 
candidate  LVAs  operate  to  deliver  weapons  on  target  and  provide  capabilities  to  the  warfighter 
(e.g.  Intelligence,  Surveillance  and  Reconnaissance  (ISR)  and  Navigation  (NAV)  satellites 
operational  in  space).  Table  3  shows  the  SAs  and  their  definitions. 
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Architecture 

Descriptions 

Current  way  of  doing  business  still  exists  in  2017 

1 

Strategic 

Constellations  consist  of  conventionally  designed  assets 

Make  use  of  DV  to  avoid  Red  OCS 

Full  responsive  launch  &  s/c  ops  capability  available 

2 

Responsive- 1 

Relatively  capable  system  already  deployed  during  peacetime 

One-to-one  replacement  for  reconstitution 

Limited  responsive  launch  &  s/c  ops  capability  available 

3 

Responsive-2 

Gap-filler,  band-aid  launch  on-need 

One-to-one  replacement  for  reconstitution 

On-orbit  servicing  capability  available 

A 

Serviceable 

Maneuver  capability  enabled  due  to  servicing 

H 

Spacecraft  have  bigger  tanks  for  maneuvering 

Responsive  launch  &  s/c  ops  capability  available 

5 

Recoverable 

Use  fly-back  (recoverable)  reusable  spacecraft 

Responsive  launch  &  s/c  ops  capability  available 

6 

Retrievable 

Use  retrievable  (by  RLV)  reusable  spacecraft 

Responsive  launch  &  s/c  ops  capability  available 

Current  way  of  doing  business  still  exists  in  2017 

7 

Storage-on-Orbit 

Constellations  consist  of  conventionally  designed  assets 

Use  spares  from  orbit  to  replace  assets  disabled  by  Red  OCS 

Table  3:  ORS  AoA  Space  Architectures 


Space  Architecture  2  (Responsive- 1)  (SA  2)  was  selected  as  the  ‘reference  architecture’  for 
the  detailed  analysis  for  the  ORS  AoA  since  it  appeared  to  be  the  most  probable  ‘world  state’  for 
an  operationally  responsive  spacelift  system. 

The  ORS  AoA  Cost  Effectiveness  Working  Group  (CEWG)  faced  the  issue  of  first 
determining  the  relationship  between  performance  and  cost;  and  second,  determining  how  to 
evaluate  the  alternatives’  performance,  cost,  and  risk  together.  The  ORS  AoA  cost  uncertainty 
analysis  addressed  a  number  of  technology  and  other  areas  of  risk.  Two  areas  of  uncertainty, 
launch  vehicle  production  cost  and  processing  time,  were  addressed  in  cost  assumptions  and 
were  significant  analysis  drivers  which  needed  to  be  bounded  as  part  of  the  regret  analysis. 

The  ICWG  made  two  significant  assumptions  concerning  these  areas  of  uncertainty  for  the 
cost  analysis  portion  of  the  study.  The  first  significant  assumption  was  that  a  50  percent 
reduction  in  launch  vehicle  production  cost  for  the  reusable  portion  of  the  launch  vehicle  was 
achievable  (50  percent  reduction  from  the  full  shuttle-like  production  cost).  The  second 
significant  assumption  was  that  a  reduction  in  launch  vehicle  processing  time  was  achievable 
(where  the  notional  best  case  was  approximately  1  to  2  days  and  the  worst  case  was 
approximately  4  to  16  days,  depending  on  the  launch  vehicle).  The  ICWG  and  CEWG  bounded 
the  trade  space  by  defining  high  and  low  production  costs,  and  faster  and  slower  processing 
times.  Figure  2  graphically  and  notionally  displays  the  wide  spread  in  production  cost  versus 
processing  time  represented  in  the  analysis.  Actual  values  of  both  axes  are  not  provided  due  to 
the  sensitivity  of  the  data. 
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Assumption  Boundaries 
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Figure  2.  Trade  Space  Bounded  by  Launch  Vehicle  Production 
Costs  and  Processing  Time  Uncertainty. 


The  ideal  ORS  architecture  should  provide  faster  launch  vehicle  processing  at  a  low  launch 
vehicle  production  cost.  If  this  is  not  achievable,  a  decision-maker  must  decide  or  settle  for  one 
of  the  other  architectures,  trading  between  cost  and  processing  time.  In  today’s  funding- 
constrained  environment,  a  low  cost  launch  vehicle  architecture  may  have  a  better  chance  of 
being  chosen  even  if  it  is  does  not  provide  faster  launch  vehicle  processing  time.  The  life  cycle 
cost  estimate  (LCCE)  for  the  LVA  will  reflect  the  lower  production  costs  offset  by  some  degree 
by  the  increased  operations  costs  cause  by  slower  processing.  This  LCCE  variability  provides  a 
dilemma  for  the  decision-maker  in  deciding  which  conditions  to  trade  off  and  for  which 
conditions  to  settle. 

Figures  3,  4  and  5  depict  the  LCCE  variability  for  the  least  expensive  LVA  of  each  type 
(Expendable,  Reusable,  Hybrid,  and  Hypersonic),  measured  for  the  low,  medium  and  high 
performance  levels  respectively.  The  cost  and  processing  time  variables  have  been  combined 
into  a  “best  case”  (faster  LV  processing  time  and  low  production  cost)  and  a  “worst  case” 
(slower  processing  time  and  high  production  cost)  in  deriving  the  LCCE  of  each  LVA 
alternative.  In  each  figure,  the  black  segment  of  each  bar  represents  the  best  case.  The  red 
portion  represents  the  additional  cost  of  the  worst  case  (slower  processing  time  and  high 
production  cost).  Other  combinations  of  the  two  variables  create  cases  whose  LCCE  fall 
between  that  of  the  best  and  worst  case.  Due  to  the  sensitive  nature  of  the  estimated  costs,  the  y- 
axis  does  not  have  the  actual  scale.  However,  the  chart  depicts  the  relative  LCCE  differences 
among  the  alternatives. 

Figure  3  illustrates  that  LVA  #71  has  the  lowest  LCCE  for  the  low  performance  level  for  SA 
2  and  under  the  best  case  (faster  LV  processing  time  and  lower  production  cost).  Figures  4  and  5 
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show  LVA  #71  also  has  the  lowest  LCCE  for  medium  and  high  performance  levels  respectively 
for  SA  2  under  the  best  case.  It  should  be  noted  that  both  LVAs  #67  and  #71  are  included  in  the 
hybrid  category.  Although  LVA  #71  appears  to  be  the  least  expensive  in  the  best  case  across 
low,  medium  and  high  performance  levels,  LVA  #67  is  a  close  second  and  has  a  lower  LCCE  in 
the  worst  case  (slower  LV  processing  time  and  high  production  cost)  in  that  LVA  category 
(hybrids). 


Cost  Sensitivity  Variability  (Low) 


- - - —  - 

■  Worst  Case  Delta  f 

■  Best  Case 

4 

g  _ 

I  1  U  1 

ELV  #4  RLV  #30  Hybrid  #67  Hybrid  #71  Hypersonic  #64 

LVA  Type 


Figure  3.  Best  and  Worst  Case  Conditions  for  Low  Performance  Level. 


The  figures  show  that  there  is  significant  variability  in  the  cost  across  the  least  expensive 
architecture  of  each  type  of  LVA.  They  also  show  that  the  cost  variability  among  alternatives 
increases  for  the  medium  and  high  performance  levels.  This  result  was  counter  intuitive:  the 
CEWG  expected  the  variability  to  be  consistent  across  performance  levels.  This  finding  led  the 
CEWG  to  examine  the  sensitivity  of  the  cost  assumptions,  and  to  determine  if  other  LVA 
alternatives  would  potentially  be  cost  effective. 
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Cost  Sensitivity  Variability  (Medium) 


ELV  #4  RLV  #30  Hybrid  #67  Hybrid  #71  Hypersonic  #64 


LVA  Type 

Figure  4.  Best  and  Worst  Case  Conditions  for  Medium  Performance  Level. 


Figures  3  through  5  graphically  depict  there  is  no  clear  ‘winner’  for  lowest  LCCE.  As  a 
result  of  the  variations  in  each  architecture  between  the  best  case  and  worst  case  costs  for  low, 
medium,  and  high  performance  levels,  the  CEWG  undertook  a  variation  of  regret  analysis  to 
shed  additional  light  on  the  issue.  Regret  analysis  is  a  subset  of  decision  analysis  and  is  a  basic 
operations  research  technique  used  to  determine  how  much  may  be  lost  if  a  different  decision 
were  chosen.  Used  here,  regret  is  measured  by  the  difference  in  LCC  between  the  optimal  (least 
expensive)  choice  and  an  alternative  within  the  same  performance  level  and  for  the  same 
conditions  (LV  production  costs  and  processing  time). 
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1 


ELV  #4  RLV  #30  Hybrid  #67  Hybrid  #71  Hypersonic  #64 

LVAType 


Figure  5.  Best  and  Worst  Case  Conditions  for  High  Performance. 


Regret 

For  the  case  study,  the  author  defined  regret  as  “the  difference  between  some  choice  and  the 
best  choice  for  a  particular  realization  of  the  uncertainties.”  Typically,  regret  occurs  when  the 
selected  alternative  is  not  the  least  expensive  of  alternative  choices  within  a  given  set  of 
assumptions  or  conditions  in  which  the  alternative  is  procured.  For  example,  System  A  may  be 
selected  because  it  appears  on  average  to  be  the  least  expensive,  but  it  is  not  the  least  expensive 
under  given  sensitivity  conditions. 

If  high  launch  vehicle  production  cost  and  slower  launch  vehicle  processing  times  actually 
occur,  there  is  regret  because  in  this  particular  circumstance  a  different  alternative  was  estimated 
to  cost  less.  Since  there  was  no  probability  for  the  chance  of  the  launch  vehicle  processing  time 
and  launch  vehicle  production  cost  variables,  two  methods  of  regret  analysis  were  selected.  To 
distil  the  vast  amount  of  data  into  meaningful  cost-effectiveness  information  for  the  decision¬ 
maker,  the  CEWG  performed  regret  analysis  on  the  average  regret  across  the  four  conditions 
within  a  performance  level  as  well  as  on  the  Minimax  regret  across  the  four  conditions  within  a 
performance  level,  for  every  performance  level  and  all  seven  SAs.  The  rest  of  this  paper  will 
discuss  the  selected  reference  SA  2. 

Average  Regret.  Average  regret  is  the  equivalent  of  having  all  equally  likely  outcomes. 
Average  regret  is  uniformly  distributed  (meaning  that  each  condition  has  the  same  probability  of 
occurring  -  which  is  the  main  and  very  significant  shortcoming  of  average  regret  -  there  is  little 
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discriminative  ability.  The  regret  function  is: 

Average  Regret  #  =  (average  (LCCE  of  other  LVA  #)  across  the  four  conditions)  -  (average 

(LCCE  of  LVA  #)  across  the  four  conditions) 

Minimization  of  the  Maximum  (Minimax)  Regret.  The  Minimax  Regret  Criterion  was 
developed  by  L.J.  Savage  and  uses  the  concept  of  opportunity  cost  to  arrive  at  a  decision.  The 
Minimax  Regret  criterion  focuses  on  avoiding  regrets  that  may  result  from  making  a  nonoptimal 
decision.  The  Savage  Minimax  regret  criterion  examines  the  regret,  opportunity  cost  or  loss 
resulting  when  a  particular  situation  occurs  and  the  payoff  of  the  selected  alternative  is  smaller 
than  the  payoff  that  could  have  been  attained  with  that  particular  situation.  The  minimax 
criterion  suggests  that  the  decision-maker  look  at  the  maximum  regret  of  each  strategy  and  select 
the  one  with  the  smallest  value.  This  approach  appeals  to  cautious  decision-makers  who  want  to 
ensure  that  the  selected  alternative  does  well  when  compared  to  other  alternatives  regardless  of 
what  situation  arises.  It  is  particularly  attractive  to  a  decision-maker  who  knows  that  several 
competitors  face  identical  or  similar  circumstances  and  who  is  aware  that  the  decision-maker’s 
performance  will  be  evaluated  in  relation  to  the  competitors.  This  criterion  is  applied  to  the 
same  decision  situation  and  transforms  the  payoff  matrix  into  a  regret  matrix. 

Minimax  regret  is  distribution  independent  (meaning  that  the  outcome  is  not  dependent  on  an 
underlying  statistical  distribution)  -  which  gives  it  its  power,  and  is  why  it  is  found  in  the 
literature  much  more  than  average  regret.  Minimax  regret  is  equivalent  to  being  pessimistic, 
“The  Eeyore  syndrome”.  For  the  Minimax  Regret  analysis  in  the  case  study,  all  other  LVAs  will 
be  compared  to  the  least  costly  LVA  within  each  performance  level.  The  Minimax  regret  value 
was  calculated  across  each  of  the  four  conditions  in  each  performance  level  for  each  SA. 
Although  these  four  conditions  do  not  have  any  probabilities  assigned  to  them,  the  analysts 
believed  they  are  not  all  equally  likely.  For  example,  obtaining  the  fast  launch  vehicle 
processing  time  and  low  launch  vehicle  production  cost  will  not  have  the  same  probability  as 
obtaining  the  slow  launch  vehicle  processing  time  and  high  launch  vehicle  production  cost. 
Decisions  made  with  minimizing  the  maximum  regret  are  to  control  the  remorse  that  inevitably 
accompanies  hindsight.  This  is  a  form  of  the  best,  worst  case  selection.  This  regret  function  is: 

Minimax  Regret  (of  a  particular  alternative  #)  =  (LCCE  of  other  LVA  #)  —  (Least  costly 
LCCE) 


Application  of  the  Regret  Analysis  Process 

The  cost  of  each  LVA  was  compared  to  one  of  the  hybrid  LVAs  to  identify  a  delta.  Four 
deltas  were  calculated  at  each  performance  level  for  the  four  conditions  given  by  launch  vehicle 
production  cost  (high  and  low)  and  launch  vehicle  processing  time  (faster  and  slower).  Then  the 
four  deltas  were  averaged  for  each  performance  level.  Launch  vehicle  processing  time  and 
launch  vehicle  production  cost  are  conditions  that  are  not  equally  likely.  Slower  launch  vehicle 
processing  time  and  high  launch  vehicle  production  cost  characterize  space  launches  today. 
However,  faster  launch  vehicle  processing  time  and  low  launch  vehicle  production  cost  are 
desired  for  the  future. 

The  regret  value  is  calculated  for  each  case  in  each  performance  level  in  each  SA.  Since 
there  are  four  conditions  for  each  performance  level  with  no  probability  assigned  to  each  case, 
the  regret  values  were  averaged  for  each  performance  level.  If  the  average  regret  value  for  a 
specific  LVA  is  positive,  then  the  least  expensive  LVA  on  average  is  better  but  if  the  average 
regret  value  is  negative,  then  that  specific  LVA  is  better.  Of  the  three  hybrid  LVAs  which  made 
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the  final  cut  of  15  LVAs,  LVA  #71  was  chosen  as  the  least  expensive  on  average  to  compare 
against  the  other  LVAs  in  the  final  cut. 

Process  to  Calculate  Average  Regret: 

The  steps  for  calculating  Average  Regret  are  pretty  straight  forward.  For  all  space 
architectures,  the  following  steps  were  performed  in  each  performance  level.  Tables  4  through  6 
illustrate  for  the  reader  the  Average  Regret  calculation  steps  for  SA2  for  the  low  performance 
level.  Only  a  portion  of  SA  2  is  displayed. 

1.  The  LCC  of  each  LVA  was  compared  to  LVA  #71  to  identify  a  delta  LCC  (in  Base  Year 
2003  millions  of  dollars  (BY03SM)).  If  the  delta  LCC  is  positive,  then  LVA  #71  is 
cheaper.  If  the  delta  cost  is  negative,  then  LVA  #71  is  more  expensive. 

2.  Four  deltas  were  calculated  at  each  performance  level  for  the  four  conditions: 

a.  Faster  Processing  Time,  High  Production  Cost 

b.  Faster  Processing  Time,  Low  Production  Cost 

c.  Slower  Processing  Time,  High  Production  Cost 

d.  Slower  Processing  Time,  Low  Production  Cost 


Architecture 

SA  2 

SA  2 

SA  2 

SA  2 

Performance 

Low 

Low 

Low 

Low 

Processing  Time 

Faster 

Faster 

Slower 

Slower 

Production  Cost 

High 

Low 

High 

Low 

Expendable  (4) 

20,679 

33,563 

16,747 

Expendable  (11) 

26,996 

39,881 

18,423 

Expendable  (12) 

37,058 

49,943 

3,374 

28,728 

Reusable  (30) 

17,876 

9,278 

37,307 

16,479 

Reusable  (36) 

34,538 

22,463 

36,488 

19,879 

Reusable  (39) 

16,488 

14,815 

15,522 

10,140 

Reusable  (41) 

21,165 

12,022 

22,335 

8,920 

Reusable  (46) 

36,394 

19,843 

72,324 

34,759 

Reusable  (47) 

39,855 

20,681 

82,208 

39,699 

Reusable  (48) 

33,320 

24,394 

24,787 

15,716 

Reusable  (51) 

27,823 

12,678 

38,152 

14,174 

Hypersonic  (64) 

55,760 

44,796 

52,728 

39,813 

Hybrid  (67) 

5,352 

14,138 

2,952 

Hybrid  (70) 

10,183 

13,502 

918 

7,996 

Hybrid  (71) 

0 

0 

0 

0 

Table  4:  Average  Regret,  Step  2,  BY03SM 


3.  The  deltas  (from  step  2)  were  summed  across  the  four  conditions. 


Architecture 

SA  2 

Performance 

Low 

Total  Regret 

(+  is  good  for  LVA  #71) 

Expendable  (4) 

62,383 

Expendable  (11) 

78,369 

12 


Expendable  (12) 

119,104 

Reusable  (30) 

80,940 

Reusable  (36) 

113,368 

Reusable  (39) 

56,965 

Reusable  (41) 

64,442 

Reusable  (46) 

163,320 

Reusable  (47) 

182,443 

Reusable  (48) 

98,217 

Reusable  (51) 

92,826 

Hypersonic  (64) 

193,097 

Hybrid  (67) 

8,695 

Hybrid  (70) 

32,599 

Table  5:  Average  Regret,  Step  3,  BY03SM 


4.  The  average  was  calculated  (sums  from  step  3  divided  by  4). 


Architecture 

SA  2 

Performance 

Low 

Average  Regret 

(+  is  good  for  LVA  #71) 

Expendable  (4) 

15,596 

Expendable  (11) 

19,592 

Expendable  (12) 

29,776 

Reusable  (30) 

20,235 

Reusable  (36) 

28,342 

Reusable  (39) 

14,241 

Reusable  (41) 

16,110 

Reusable  (46) 

40,830 

Reusable  (47) 

45,611 

Reusable  (48) 

24,554 

Reusable  (51) 

23,207 

Hypersonic  (64) 

48,274 

Hybrid  (67) 

2,174 

Hybrid  (70) 

8,150 

Table  6:  Average  Regret,  Step  4,  BY03SM 


5.  If  the  average  delta  is  positive,  then  LVA  #71  is  cheaper. 

For  low  performance  level,  LVA  #71  clearly  has  the  least  regret  since  all  other  values  are 
positive.  For  the  medium  and  the  high  performance  levels,  there  are  a  few  negative  regret 
figures  indicating  that  LVA  #71  is  not  the  best  choice.  Table  7  shows  the  Average  Regret  results 
for  each  performance  level  for  the  reference  architecture  SA  2. 
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Architecture 

SA  2 

SA  2 

SA  2 

Performance 

Low 

Medium 

High 

Average  Regret  versus  LVA  #71 

(+  is  good  for  LVA  #71) 

Expendable  (4) 

15,596 

11,727 

Expendable  (11) 

19,592 

18,455 

4,764 

Expendable  ( 1 2) 

29,776 

27,023 

4,299 

Reusable  (30) 

20,235 

50,253 

58,743 

Reusable  (36) 

28,342 

62,334 

54,426 

Reusable  (39) 

14,241 

66,073 

89,367 

Reusable  (41) 

16,110 

47,311 

54,096 

Reusable  (46) 

40,830 

62,298 

62,382 

Reusable  (47) 

45,611 

64,472 

52,762 

Reusable  (48) 

24,554 

87,401 

108,274 

Reusable  (51) 

23,207 

86,647 

98,980 

Hypersonic  (64) 

48,274 

153,307 

228,433 

[Hybrid  (67) 

2.174, 

-496 

.  1 0.6KK 

Hybrid  (70) 

8,150 

5,974 

•  1 ,43  ^ 

Table  7:  Average  Regret,  SA  2  Results,  BY03SM 


As  mentioned  earlier,  the  issue  or  concern  arising  with  this  method  is  that  each  condition  for 
the  LVA  estimates  does  not  have  the  same  probability. 

However,  consider  the  question,  “In  which  instances  is  LVA  #71  not  the  most  cost  effective 
answer?”  The  average  hides  the  results  for  those  conditions  where  hybrid  does  not  win.  The 
main,  and  very  significant,  shortcoming  of  average  regret  is  the  assumption  that  all  outcome 
states  are  equally  likely.  There  is  no  evidence  they  are  equally  likely  and  plenty  of  evidence  that 
they  are  not  equally  likely.  The  distribution  is  skewed  toward  the  least  favorable  outcome:  high 
launch  vehicle  production  cost  and  slower  launch  vehicle  processing  time. 

Process  to  Calculate  Minimax  Regret  for  the  ORS  AoA 

For  all  space  architectures,  the  following  steps  were  performed  in  each  performance  level. 
Tables  8  and  9  are  shown  to  assist  the  reader  in  following  the  Minimax  Regret  calculation 
process.  Only  a  portion  of  SA  2  is  displayed. 

1 .  The  lowest  LCCE  was  found  for  each  of  the  four  conditions: 

a.  Faster  Processing  Time,  High  Production  Cost 

b.  Faster  Processing  Time,  Low  Production  Cost 

c.  Slower  Processing  Time,  High  Production  Cost 

d.  Slower  Processing  Time,  Low  Production  Cost 

2.  The  regret  value  was  calculated  by  subtracting  the  lowest  LCCE  (from  step  1)  from 

each  of  the  other  LVAs  within  the  respective  condition 


Architecture 

SA  2 

SA  2 

SA  2 

SA  2 

Performance 

Low 

Low 

Low 

Low 

Processing  time 

Faster 

Faster 

Slower 

Slower 

Production  Cost 

High 

Low 

High 

Low 

Expendable  (4) 

20,679 

33,563 

5,141 

16,747 

14 


Expendable  (11) 

26,996 

39,881 

6,817 

18,423 

Expendable  (12) 

37,058 

49,943 

17,122 

28,728 

Reusable  (30) 

17,876 

9,278 

51,055 

16,479 

Reusable  (36) 

34,538 

22,463 

50,236 

19,879 

Reusable  (39) 

16,488 

14,815 

29,269 

10,140 

Reusable  (41) 

21,165 

12,022 

36,083 

8,920 

Reusable  (46) 

36,394 

19,843 

86,072 

34,759 

Reusable  (47) 

39,855 

20,681 

95,956 

39,699 

Reusable  (48) 

33,320 

24,394 

38,535 

15,716 

Reusable  (51) 

27,823 

12,678 

51,900 

14,174 

Hypersonic  (64) 

55,760 

44,796 

66,475 

39,813 

Hybrid  (67) 

5,352 

14,138 

0 

2,952 

Hybrid  (70) 

10,183 

13,502 

14,666 

7,996 

Hybrid  (71) 

0 

0 

13,748 

0 

Table  8:  Minimax  Regret,  Step  2,  BY03SM 


3.  Then  the  maximum  regret  value  for  each  LVA  was  determined  by  taking  the  largest 
regret  value  of  the  four  conditions _ 


Architecture 

SA  2 

Performance 

Low 

Expendable  (4) 

33,563 

Expendable  (11) 

39,881 

Expendable  (12) 

49,943 

Reusable  (30) 

51,055 

Reusable  (36) 

50,236 

Reusable  (39) 

29,269 

Reusable  (41) 

36,083 

Reusable  (46) 

86,072 

Reusable  (47) 

95,956 

Reusable  (48) 

38,535 

Reusable  (51) 

51,900 

Hypersonic  (64) 

66,475 

Hybrid  (67) 

14,138 

Hybrid  (70) 

14,666 

Hybrid  (71) 

13,748 

Table  9:  Minimax  Regret,  Step  3,  BY03SM 


4.  Lastly,  the  minimum  of  the  maximum  regret  values  for  was  determined  in  each 
performance  level 

The  Minimax  Regret  analysis  results  for  SA  2  are  shown  in  Table  10. 
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Architecture 

SA  2 

Performance 

Low 

Medium 

High 

Max  Regret 

Expendable  (4) 

33,563 

39,004 

35,574 

Expendable  (11) 

39,881 

47,482 

47,305 

Expendable  (12) 

49,943 

56,713 

48,609 

Reusable  (30) 

51,055 

130,510 

185,371 

Reusable  (36) 

50,236 

133,708 

155,299 

Reusable  (39) 

29,269 

146,831 

233,414 

Reusable  (41) 

36,083 

116,009 

172,178 

Reusable  (46) 

86,072 

145,051 

187,384 

Reusable  (47) 

95,956 

150,581 

171,974 

Reusable  (48) 

38,535 

178,474 

257,857 

Reusable  (51) 

51,900 

200,298 

267,315 

Hypersonic  (64) 

66,475 

281,238 

478,988 

Hybrid  (67) 

14,138 

14,777 

21,107 

Hybrid  (70) 

14,666 

23,869 

47,700 

Hybrid  (71) 

13,748 

26,226 

64,576 

Mini-Max  Regret 

Hybrid 

(71) 

Hybrid 

(67) 

Hybrid 

(67) 

13,748 

14,777 

21,107 

Delta  (LVA  #67  -  LVA  #71) 

-390 

11.449 

43.468 

Table  10:  Minimax  Regret,  SA  2  Results,  BY03SM 


It  is  important  to  examine  the  first  and  second  choices  for  inconsistencies.  Looking  at  the 
Minimax  numbers,  LVA  #71  only  wins  for  the  low  performance  case  ($13,748M),  and  LVA  #67 
($14,1 3  8M)  comes  in  as  a  close  second.  In  the  medium  performance  case,  LVA  #67  ($14,777M) 
is  the  clear  winner  while  LVA  #71  has  the  greatest  regret  ($26,226M)  within  hybrids.  The  same 
is  true  only  more  so  at  the  high  performance  level.  In  fact,  at  the  high  performance  level  every 
expendable  has  less  regret  than  LVA  #71.  In  the  high  performance  level,  hybrid  LVA  #67  is  the 
clear  winner  at  $21,107M;  expendable  LVA  #4  is  next  at  $35,574M,  then  expendable  LVA  #1 1 
at  $47,305M. 

Upon  comparing  LVA  #67  and  LVA  #71,  the  last  row  of  Table  10  shows  the  minimax  regret 
delta  between  the  two.  When  LVA  #71  is  the  clear  winner,  the  delta  is  negative  when  compared 
to  LVA  #67.  This  happens  only  for  the  low  performance  case.  The  medium  and  high 
performance  cases  clearly  show  that  LVA  #67  has  less  regret  since  those  numbers  are  positive. 

Comparison  of  Average  Regret  versus  Minimax  Regret 

Average  Regret  hides  results  for  the  conditions  where  hybrid  does  not  win.  For  low 
performance  hybrid  LVA  #71  had  the  least  regret  on  average.  However,  for  medium  and  high 
performance,  hybrid  LVA  #67  has  the  least  regret  on  average.  We  do  not  have  insight  into  the 
combination  of  conditions  (cost  and  processing  time)  for  which  LVA  #71  is  better  than  LVA 
#67;  therefore  we  do  not  know  which  conditions  directly  affected  the  LCCEs  of  LVA  #67  and 
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LVA  #71 .  This  led  the  CEWG  to  Minimax  Regret  analysis. 

Minimax  regret  revealed  that  the  hybrid  type  launch  vehicle  had  the  least  regret  for  all  levels 
of  performance.  Through  Minimax  regret  analysis,  the  decision-maker  can  see  which  hybrid 
launch  vehicle  to  pursue  if  growth  in  performance  (low,  medium,  or  high  utility),  funding,  or  risk 
were  priorities.  For  low  performance  LVA  #71  has  the  least  regret  ($13,748M),  but  for  medium 
and  high  performance  LVA  #67  has  the  least  regret  ($14,777M  and  $21,107M  respectively). 

The  existing  condition  of  our  country’s  ability  for  ORS  is  that  we  currently  have  slower 
launch  vehicle  processing  time  (more  than  48  hours)  and  high  launch  vehicle  production  cost. 
The  ideal  condition  is  faster  launch  vehicle  processing  time  (24  to  48  hours)  and  low  launch 
vehicle  production  cost.  The  distribution  of  conditions  is  skewed  toward  the  least  favorable 
outcome:  high  launch  vehicle  production  cost,  slower  launch  vehicle  processing  time.  That  is, 
the  probability  of  high  launch  vehicle  production  cost  and  slower  launch  vehicle  processing  time 
is  greater  than  the  probability  of  low  launch  vehicle  production  cost  and  faster  launch  vehicle 
processing  time: 

P  (high  production  cost  &  slower  processing  time)  >  P  (low  production  cost  &  faster  processing  time) 

The  data  do  not  seem  to  support  the  conclusion  that  LVA  #71  is  the  best.  LVA  #71  wins  on 
average  regret  only  at  the  low  performance  case,  with  LVA  #67  a  close  second.  LVA  #71  is  tied 
with  LVA  #67  at  medium  performance,  and  loses  to  LVA  #67  (and  LVA  #4)  at  high 
performance.  Choosing  LVA  #71  ignores  indications  that  LVA  #67  is  a  more  robust  choice, 
especially  if  a  decision-maker  wishes  to  procure  a  system  with  the  potential  to  increase 
performance.  If  LVA  #71  is  somehow  better  from  a  demonstrator  (or  political)  point  of  view,  no 
evidence  of  that  is  presented  here.  All  of  this  is  evidence  indicating  the  lack  of  information  for  a 
specific  hybrid-type  vehicle  to  make  a  sound  acquisition  decision. 

Further  analysis  needs  to  be  accomplished  to  get  a  better  picture  of  risk  and  regret  for  an 
operationally  responsive  launch  system.  An  operationally  responsive  hybrid  vehicle 
demonstration  would  reduce  cost  and  operability  uncertainties  as  well  as  technical  risk.  The 
ORS  AoA  recommendation  was  to  pursue  a  first  stage  reusable  launch  vehicle  with  expendable 
upper  stages  demonstrations  to  assess  cost,  operability,  technology  readiness  levels;  to  test 
operations  concepts/processes;  and  to  assess  scalability  of  cost  and  operations  data  and  test  more 
realistic  operations  concepts. 

Since  completion  of  this  study,  the  HQ  AFSPC  Commander  gave  permission  in  January  2005 
for  the  ORS  Demonstrator  to  begin.  A  Phase  I  Program  Research  and  Development 
Announcement  (PRD A)  notice  was  posted  on  February  16,  2005  for  the  Affordable  REsponsive 
Spacelift-Subscale  Demonstration  (ARES-SD).  The  ARES  Phase  1  Concept  Development  and 
Demonstration  Planning  Program  is  contracted  by  the  Department  of  the  Air  Force,  Air  Force 
Space  Command,  SMC  -  Space  and  Missiles  System  Center. 

Summary 

In  closing,  this  paper  defined  the  differences  between  risk  and  regret.  Although  risk  has  no 
one  definition,  some  theorists  have  defined  general  methods  to  assess  risk  as  an  expected  after- 
the-fact  level  of  regret.  Two  types  of  regret  analysis,  Average  Regret  and  Minimax  Regret,  were 
demonstrated  using  the  HQ  AFSPC  ORS  AoA  as  a  case  study.  Average  Regret  was  a  very  high 
level  analysis  while  Minimax  Regret  ‘peeled  back  the  onion’  a  little  further.  Minimax  Regret 
analysis  improves  the  chance  making  the  right  decision,  achieving  a  more  positive  outcome. 
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Since  there  is  not  enough  detailed  information  to  determine  if  faster  processing  time  and  low 
production  cost  are  achievable  assumptions,  the  ORS  AoA  team  recommended  that  an 
operationally  responsive  hybrid  vehicle  demonstration  would  reduce  cost  and  operability 
uncertainties  as  well  as  technical  risk.  The  bottom  line:  different  tools  and  types  of  analysis 
yield  different  analysis  results.  To  achieve  unbiased  analysis,  the  analyst  cannot  overlook  one 
tool  or  type  of  analysis  over  another.  When  used  appropriately,  Average  Regret  and  Minimax 
Regret  can  be  powerful  analytical  tools  if  limitations  of  each  are  known  and  understood. 

Disclaimer 

The  views,  opinions,  and/or  findings  contained  in  this  report  are  those  of  the  author  and 
should  not  be  construed  as  an  official  Government  position,  policy,  or  decision,  unless 
designated  by  other  documentation.  No  classified  documents  or  sources  were  referenced  in  the 
preparation  of  this  paper. 


Acronyms 


AFSPC/DR 

HQ  AFSPC  Directorate  of  Requirements 

ARES-SD 

Affordable  REsponsive  Spacelift-Subscale  Demonstration 

BY03$M 

Base  Year  2003  millions  of  dollars 

CEWG 

Cost  Effectiveness  Working  Group 

HQ  AFSPC 

Head  Quarters  Air  Force  Space  Command 

ICWG 

Integrated  Concepts  Working  Group 

ISR 

Intelligence,  Surveillance  and  Reconnaissance 

LCC 

life  cycle  cost 

LCCE 

life  cycle  cost  estimate 

LVAs 

launch  vehicle  architectures 

Minimax 

Minimization  of  the  Maximum 

MOEs 

Measures  of  Effectiveness 

MOPs 

measures  of  system  performance 

NAV 

Navigation 

ocs 

Offensive  Counterspace 

ORS 

Operationally  Responsive  Spacelift 

PRDA 

Program  Research  and  Development  Announcement 

SAs 

Space  Architectures 
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